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Abstract 


The Lumina Degree Qualifications Profile (DQP): Implications for Assessment 


In January 2011, the Lumina Foundation published its Degree Qualifications Profile (DQP) to chal- 
lenge faculty and academic leaders in the U.S. to think deeply and concretely about aligning expec- 
tations for student learning outcomes across higher education. Since then, the DQP has kindled 
extensive discussions about what the postsecondary degrees granted by American colleges and univer- 
sities really mean with respect to what graduates know and can do. But the text of the DQP itself 
provides only limited guidance to stakeholders with respect to assessment. 


In order to render the Profile’s potential real, institutions and their faculties will need to develop 
consistent and systematic ways to gather evidence that the competencies that the DQP describes are 
actually being mastered at the levels claimed. In this paper, I explore some of what needs to be done 
in this area and provide a few tools and techniques (some of which are already in widespread use) that 
may help us move forward. In offering them, I invite faculties at all our colleges and universities to 
carefully examine what the DQP asks us to do in designing more aligned and integrated approaches 
to teaching, learning, and determining student competence—as well as to actively experiment with 
these ideas and techniques with their colleagues. 
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Foreword 


George Kuh & Stan Ikenberry 


Assessment and the DQP: A Brave New World Beckons 


Shortly after we launched the National Institute for Learning Outcomes Assessment (NILOA) in 
2009, Lumina Foundation for Education announced its Big Goal: an increase, by 2025, from 40% 

to 60% in the proportion of adults in the United States with a postsecondary credential or degree. 
Because it was soon clear that achieving this landmark would be a hollow attainment if the quality of 
the learning was substandard, the nonnegotiable target became high-quality credentials in the form of 
degrees and certificates with well-defined and transparent learning outcomes that provide clear path- 
ways to employment and further education. 


Colleges and universities were already familiar with “learning outcomes” language, having been 
pressed by both regional and specialized accreditors for almost a decade to stipulate in concrete terms 
what students should know and be able to do and, further, to demonstrate the extent to which their 
students achieved those goals. In fact, by 2010, surveys by the Association of American Colleges and 
Universities (AAC8&U) and NILOA found that more than three quarters of all institutions had devel- 
oped outcomes statements to guide teaching and represent student learning. Proving far more difficult 
for institutions was moving from lofty, broad, and sometimes vague descriptions of student perfor- 
mance to demonstrable evidence that students had, in areas institutions deemed appropriate, indeed 
become proficient. Getting more faculty involved in assessing student learning was a major challenge, 
as NILOA’s 2009 survey had shown. Moreover, a foreboding prevailed, fueled in part by the Spellings 
Commission deliberations and studies of undergraduate student achievement claiming that great vari- 
ation existed, both in institutions’ expectations for their students as well as in students’ performance. 
These and other factors prompted the call for greater clarity about what credentials and degrees actu- 
ally represent with regard to student attainment. Lumina Foundation’s Degree Qualifications Profile 
(DQP) is intended to help colleges and universities address this formidable challenge. 


In this thoughtful paper, Peter Ewell explains why the DQP was needed to clarify and describe in 
behavioral terms what students should know and be able to do at various degree levels, and how 

the DQP comports with other recent efforts—such as AAC&U’s Essential Learning Outcomes—to 
articulate what the world needs today from its college-educated populace. The DQP says very little, 
however, as Peter points out, about assessment approaches that would be sufficient to document the 
skills and competencies expected of students earning associate’s, bachelor’s, and master’s degrees. In 
fact, one of NILOA’s major interests in the DQP is to discover what institutions are doing by way of 
assessment within the DQP framework and to share the most the promising of these practices across 


the field. 


Forcefully underscoring many of Peter’s key points in the Afterword, Carol Schneider persuasively 
argues that for the DQP to realize its potential as a bold reform vehicle, colleges and universities have 
to address two nontrivial challenges. The first is their all-too-common reaction to calls for systemic 
cultural and operational changes: to hunker down and hold fast to familiar ways of understanding 
and behaving. The second is their tendency to examine students’ learning experiences in piecemeal 
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Foreword (continued) 


fashion, thereby missing a signature DOP emphasis: integrating student attainment across the 

My! Bese P. ae 
postsecondary years in ways that encompass what students gain from general education, the major 
program, and out-of-class experiences. 


While colleges and universities from one end of the country to the other are experimenting with the 
DQP, addressing the assessment challenges in implementing the DQP is essential to support and 
advance their work and, ultimately, to ensure that students are getting what they need from postsec- 
ondary education. Peter Ewell and Carol Schneider are the perfect pair to take on this task. Both 
helped craft the DQP. For decades, both have championed the importance of rigorous assessment 
of student learning to assure and demonstrate collegiate quality. And both are thought leaders— 
informing and inspiring as they challenge and enable. 


We are delighted to introduce this Occasional Paper, NILOA’s 16" and commend it to your attention. 
We are greatly indebted to Peter and Carol for accepting our invitation to help us understand what 
assessing student learning might look like within the DQP framework. 


George Kuh 

Director, National Institute for Learning Outcomes Assessment 
Adjunct Professor, University of Illinois 

Chancellor’s Professor Emeritus, Indiana University 


Stan Ikenberry 


Co-Principal Investigator, National Institute for Learning Outcomes Assessment 
President Emeritus and Regent Professor, University of Illinois 
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The Lumina Degree Qualifications Profile 


(DQP): Implications for Assessment 


Peter Ewell 


In January 2011, Lumina Foundation published its Degree Qualifications 
Profile (DQP) to challenge faculty and academic leaders in the U.S. to 

think deeply and concretely about aligning expectations for student learning 
outcomes across higher education. Together with Cliff Adelman, Paul 
Gaston, and Carol Geary Schneider, I was privileged to be one of the four 
primary authors of the DQP. Since it was released, the DQP has kindled 
extensive discussions about what postsecondary degrees granted by American 
colleges and universities really mean with respect to what graduates know 
and can do. With respect to assessment, however, the text alone of the DQP 
provides stakeholders only limited guidance. 


To exploit the Profile’s full potential, institutions and their faculties need 
to develop consistent and systematic ways to gather evidence that the 
competencies the DQP describes are actually being mastered at the levels 
claimed. I prepared this NILOA Occasional Paper to explore some of what 


needs to be done in this area and to provide a few tools and techniques (some 


of which are already in widespread use) that may help us move forward. 
In offering these, I invite faculties at all of our colleges and universities to 
carefully examine what the DQP asks us to do in designing more aligned 
and integrated approaches to teaching, learning, and determining student 
competence—as well as to actively experiment with these ideas and 
techniques with their colleagues. 


Why the DQP? 

Like its counterpart “qualifications frameworks” in other nations, the DQP 
attempts to establish specific learning expectations for graduates receiving a 
particular degree. The Profile proposes sets of competencies in five areas of 
student learning—Specialized Knowledge, Broad Integrative Knowledge, 
Intellectual Skills, Applied Learning, and Civic Learning—and addresses 
three degree levels—associate’s, bachelor’s, and master’s. At each degree 
level, the competencies themselves are described in terms of “action verbs” 
that portray what a student at that level can actually do. For example, 
competencies at the associate’s level may require students to “describe” 

or “present” a topic, at the bachelor’s level to “construct” or “explain” 
something, and at the master’s level to “create” or “assess” something.! 


These “action verbs” provide important initial guidance to faculty, as I argue 
later, in constructing appropriate assignments and examination questions 

to determine student mastery. The language itself of the DQP’s competency 
statements also frequently provides guidance for assessment because, for 
each competency, it suggests the kinds of demonstrations that might be 
proper—for instance, a research paper, a class project, or a performance. 

As DQP authors, we believe strongly that the curriculum at any institution 


‘The resulting hierarchy can be characterized as loosely analogous to “Bloom’s Taxonomy,” a 
familiar description of student abilities arrayed in an ascending order of cognitive complexity 


(Bloom, 1956). 


The Profile proposes sets of 


competencies in five areas of student 


learning—Specialized Knowledge, 
Broad Integrative Knowledge, 
Intellectual Skills, Applied Learning, 
and Civic Learning—and addresses 
three degree levels—associate’s, 
bachelor’s, and master’s. 
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should simultaneously ensure all students adequate opportunities to study 
an area in depth as well as to master topics that cut across the DQP’s five 
domains. There is plenty of evidence that mastery of these five areas is critical 
if graduates are to be productive in the workforce and effective as citizens.” 
As a result, although we present each of the five areas of student learning 
independently, we also believe they interact with respect to both mastery and 
application. For example, good oral and written skills decisively condition 

an individual’s ability to “develop and justify a position on a public issue,” 

as called for in Civic Learning (Lumina Foundation, 2011, p.16). Students 
must demonstrate that they can apply their learning in a variety of settings 
and solve problems spanning a wide range of academic subjects and societal 
settings. 


The central message of the DQP is intentionality. Intentionality should 
govern the goals we develop to define our degrees, the curricula and 
pedagogies we design and deploy to make the goals real, and the assessments 
we use to determine if we have been successful. This message is not new. 
Indeed, many of these points were made at the dawn of the assessment 
movement in such bellwether reports as Involvement in Learning (National 
Institute of Education, 1984) and Integrity in the College Curriculum 
(Association of American Colleges, 1985). American higher education has 
been making steady progress on this journey since then, and colleagues at 
many colleges and universities have contributed. If embraced creatively 
and sincerely, the DQP provides a rare opportunity, we believe, to go even 
farther—not just in assessment but in many other aspects of teaching and 
learning as well. 


Some Assessment Implications 

DQP competencies are offered as statements of mastery, not aspiration. 
They describe what every graduate of a degree program at a given level 
ought to know and be able to do. Regardless of the specific content of 
these competencies, which will naturally differ among institutions and 
overlap only partially with those contained in the DQP itself, this posture 
of universal expectation represents something of a departure from the way 
most U.S. colleges and universities currently set learning outcomes goals and 
conduct assessment. Current assessment practice, for the most part, rests on 
faculty-established goals, developed independently at each institution, for 
what graduates should know and be able to do. 


Whether or not graduates attain these goals is then investigated on average 
by using various methods to examine the performance of representative 
samples of students.* Only a few institutions now require a culminating 
demonstration of mastery as a condition of graduation. 


*For example, see http://www.aacu.org/leap/documents/2008 Business Leader _Poll.pdf 
3Alverno College and similar mastery-based institutions like Excelsior University and Western 
Governors University are prominent exceptions to this pattern, as they require all students to 
demonstrate mastery of all outcomes as a condition of graduation. 


Intentionality should govern the goals 


we develop to define our degrees, 

the curricula and pedagogies we 
design and deploy to make the goals 
real, and the assessments we use to 
determine if we have been successful. 
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This dominant approach to examining academic quality reflects one of the 
central concerns during the mid-1980s, when the assessment movement 
began—that faculty-awarded grades cannot provide valid and reliable 
evidence of student academic achievement because they themselves are 

of uncertain reliability and validity. As a result, purpose-built assessment 
approaches designed to meet the canons of educational outcomes 
measurement were needed (Ewell, 2002). As a supplement to grades, these 
approaches were generally added on to the existing teaching and learning 
process. At the outset of the assessment movement, such “exoskeletal” authority. 
approaches were predominantly standardized tests. In the subsequent two 

decades, however, standardized examinations were gradually supplanted 

by more “authentic” assessment methods like capstone experiences or 

culminating demonstrations, portfolios comprised of student work products, 

or rubric-based ratings of class assignments. Despite this positive evolution, 

however, the underlying philosophy of assessment at most institutions, at 

least for purposes of external accountability, has centered largely on periodic 

inspection of samples of students. 


The kind of assessment that the 

DQP invites keeps faculty judgment 
at the center of assessment, rather 
than surrendering the certifying of 
student mastery to an external test or 


By insisting that all graduates master all of the described competencies, the 
DQP implies a significant shift in the underlying philosophy of assessment. 
In place of evidence-gathering activities added on to the teaching and 
learning process to “check up” on its effectiveness, assessment activities are 
embedded within the process in the form of progressively more challenging 
exercises, performances, and assignments for demonstrating student mastery 
at multiple points. The kind of assessment that the DQP invites, at the 

same time, keeps faculty judgment at the center of assessment, rather than 
surrendering the certifying of student mastery to an external test or authority. 
To be sure, this kind of assessment demands collective discipline and hard 
collaborative work from faculty to develop assignments and examination 
questions up to the task of determining student competence. Unexamined, 
untrammeled academic freedom, in this context, is out of place. Yet rather 
than fear some form of standardization, faculty will do well to remember that 
in designing assessment within the DQP they retain as much responsibility 
as in any other aspect of the curriculum. 


This shift in underlying philosophy has a number of concrete implications 
for assessment practice. In separate subsections of this paper, I flesh out 
some of these implications in more detail and provide examples of particular 
techniques and approaches. While many of these, to be sure, have been 
developing for some time, accelerating these developments are ongoing 
DQP-associated activities—especially those in the purview of Lumina’s 
follow-on grants enabling institutions to experiment with the DQP. 


Curricular Mapping 

An important prerequisite to determining the extent to which students 
are mastering the various DQP competencies is obtaining a clear picture 
of where and how particular competencies are expected, enhanced, or 
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assessed across the courses that constitute a given curriculum. The best way 
to accomplish this is to construct a “map” of the institution’s curriculum 
that clearly identifies intersections between course content and the DQP 
competencies. Sometimes called an “alignment matrix,” such an exercise 

has become increasingly visible in institutional assessment practice over the 
last decade. Curricular mapping has been especially common in general 
education where the coherence of the curriculum is rarely otherwise checked 
(Allen, 2006; Driscoll & Wood, 2007), but it has also become a feature 

of assessing individual academic programs (Allen, 2004) and even student 
affairs programs (Bresciani, Gardner, & Hickmott, 2009). 


At its most straightforward, a “curriculum map” is a two-dimensional matrix 
that represents individual courses on one dimension and competencies on 
the other. Entries within each cell can be constructed to communicate many 
things including 


¢ Whether or not the competency is taught at the college level in the 
course. For example, a typical freshman-level English composition 
course would presumably address the DQP competency “presents 
substantially error-free prose in both argumentative and narrative 
forms to generalized and specific audiences” (Lumina Foundation, 
2011, p.14). Entries here might further distinguish the level 
at which the competency is taught or reinforced—for example 
introductory, intermediate, or degree-level.* 


¢ — The level of proficiency in the competency required to effectively engage 
course material. This is, essentially, the level of proficiency that 
should be expected as prerequisite to passing the course. 


© Whether or not the student’ proficiency in the competency is directly 
tested or evaluated as part of the course. Many ways to assess 
competencies will likely be present including examinations, projects, 
assignments, field placements and internships, or other direct 
demonstrations. Most institutions using mapping also indicate the 
specific mode of demonstration used and how the resulting level of 
proficiency is coded or described. 


¢ = The level of proficiency in the competency at which the passing student 
exits the course. Most institutions that use curriculum maps have 
established several levels of proficiency for key learning outcomes 
that are below the three degree levels included in the DQP. 


‘This is consistent with (AAC&U, 2008) and best illustrated by the assessment framework of 


the University of Charleston, an active member of the Council of Independent Colleges (CIC) 


DQP Consortium (see http://www.ucwv.edu/faculty_center/). 


An important prerequisite to 
determining the extent to which 
students are mastering the various 
DQP competencies is obtaining 

a clear picture of where and how 
particular competencies are expected, 
enhanced, or assessed across the 
courses that constitute a given 
curriculum. 
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° Whether or not the passing student demonstrates proficiency in the 
competency at a level corresponding to degree mastery (¢.g., associate’, 
bachelor’, or master’s). This, of course, is the ultimate application 
of assessment in connection with the DQP. Achieving the requisite 
level of proficiency through course examinations, assignments, or 
other demonstrations—frequently in the context of a capstone 
experience—constitutes certification that the student has mastered 
the competency and can be awarded the degree. 


For purposes of the DQP, mapping is usually done both for general 
education courses and for some courses in each major field, beginning with 
courses most commonly taken. Creating a curriculum map generally occurs 
under the direction of a multidisciplinary committee or task force comprised 
of representatives drawn from a wide array of academic programs and 
student affairs offices. Typically, the committee requests the faculty member 
of record for each course to fill out a standard template for the course that 
records how the course actualizes each subdomain of the DQP in each of the 
five competency domains. 


Figure 1, prepared for a DQP demonstration project currently under way at 
nine Historically Black Colleges and Universities (HBCUs) in the Southern 
Association of Colleges and Schools (SACS) accreditation region, with 
funding from Lumina, provides an example of such a template. Entries 

in each cell indicate the mechanism through which student proficiency is 
demonstrated, such as an examination, assignment, project, performance, 
and so forth. Maps for courses with multiple sections can be prepared by the 
lead faculty member, so long as all sections of the course address the same 
content and contain comparable assessments. 


A Course Level Curriculum Map 


Intellectual Skills - Bachelors Level 


; Use of , F vee 
Analytical ; Engaging Diverse | Quantitative 
Competency Information 
Inquiry Perspectives 
Resources 


Course #1 (ex. ENGLISH 101) 
Is it addressed? 


How is it tested or assessed? 
Course #2 
Is it addressed? 


How is it tested or assessed? 


Figure 1. Course Level Curriculum Map Rubric 


A “curriculum map” is a two- 
dimensional matrix that represents 
individual courses on one dimension 
and competencies on the other. 


Communications 
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The resulting map provides an invaluable aid for identifying gaps in 
curricular coverage with respect to DQP competencies, as well as for 
planning where particular assignments or other demonstrations should be 
located. For example, Westminster College, Nebraska Methodist College, 
and McKendree University—all participants in the Council of Independent 
Colleges (CIC) DQP project—are using curriculum mapping to determine 


Mapping is usually done both for 
general education courses and 
for some courses in each major 


the overlap between their own student learning outcomes statements and the field, beginning with courses most 


DQP competencies in order to improve course coverage of each domain.’ commonly taken. 
In the demonstration project launched by the Higher Learning Commission 
(HLC), Marshall University is using a similar mapping process to evaluate 
overlap and to guide changes in the wording of their own outcomes 
statements. Meanwhile, Brandman University is using a dual mapping 
approach to the DQP and to the broadly similar Association of American 
Colleges and Universities (AAC8&U) Essential Learning Outcomes as part 
of its accreditation process with the Western Association of Schools and 
Colleges (WASC) Senior Commission, as well as part of its effort design a 
new general education program.° Finally, fellow WASC project participant 
at The Master’s College created a variety of mapping templates designed to 
help faculty document various ways established competencies are addressed 
in each course. 


Other institutions working with the DQP have mapped activities and 
instructional good practices as well as competencies. For example, several 
colleges participating in the CIC DQP project have identified and mapped 
learning activities intended to foster student development in the DQP 
domain of Broad Integrative Knowledge. Coupled with data about student 
course-taking patterns, the results enable these colleges to determine the 
extent to which students are participating in these activities and where 
changes might be needed to ensure that they are. In a similar effort, San Jose 
State University employed a technique called “institutional effort mapping” 
to determine and coordinate campus activities promoting diversity both 
within and beyond the curriculum (Halualani, Haiker, & Lancaster, 2010). 


While curricular mapping is necessary and critical in implementing an 
intentional curriculum, however, it is only a first step. Too many institutions 
now engaged with the DQP stop short of the difficult work of developing 
the needed assignments, examination questions, and projects that enable the 
collection of meaningful evidence of student mastery. 


>Examples noted throughout this paper that lack specific citations were obtained through 
NILOA’ ongoing efforts to “harvest” lessons of the many follow-on grants funded by Lumina 
to demonstrate applications of the DQP (see Kuh, 2012). 

®See https://www.wascsenior.org/files/Brandman%20 University%20Adopts%20the%20 
Degree%20Qualification%20Profile January%2031%202012 final.pdf. 
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Competency Requires Action 

Competency statements in the DQP are deliberately and relentlessly couched 
in “action verbs” that describe what students at particular levels should be 
expected to do. As DQP authors, we avoided using language like “know,” 
“appreciate,” or “value” that is common in institutional mission statements 
in favor of language signifying activity or production. We did so because this 
kind of language points directly toward aspects of pedagogy that can actually 
be observed and toward the production of student artifacts that can actually 
be evaluated. 


Assignments or examination questions designed to determine proficiency 

in particular DQP competencies, consequently, must require students to 
generate a product of some kind—a research paper, an oral presentation, a 
dance performance, a translation of a text from one language to another, an 
engineering design, and so forth. Merely identifying a “correct” answer from 
a set of posed alternatives is not a production task. Because the assessments 
associated with DQP competencies require students to directly demonstrate 
mastery, the assessment really is the competency from an operational 
standpoint. This is not the case for assessments, like multiple-choice 
examinations or survey self-report items, where possession of the competency 
must be indirectly inferred from a respondent’s answers. 


As DQP authors, following the description of a DQP competency, we 
frequently provided examples of the kind of demonstration that would 

be appropriate to show proficiency. We intend these to be illustrative, not 
exhaustive. For every competency, many different kinds of demonstrations 
can be envisioned. Probably the most frequent type of example is a piece of 
writing generated by a student in response to a posed topic or question. Most 
of the writing examples we offered in the DQP go well beyond exposition or 
description to specify particular attributes of the written product that should 
be present—such as competing hypotheses, multiple forms of evidence, and 
proper citation of sources. But many of the most interesting examples invoke 
other forms of demonstration—such as building or producing a physical 
product or performance. Prominent in Civic Learning, for instance, is 
“collaborates with others in developing and implementing an approach to a 
civic issue” (Lumina Foundation, 2011, p.16). 


The key to determining the adequacy of a given assignment to demonstrate 
mastery is to revisit the action verbs that describe the competency itself. If 
the task posed by the assignment does not require the student to concretely 
perform the required action, it is not adequate. For example, a competency at 
the bachelor’s level for Use of Information Resources under Intellectual Skills 
reads “incorporates multiple information resources presented in different 
media and/or different languages...with citations in forms appropriate 

to those resources, and evaluates the reliability and comparative worth 

of competing information resources” (Lumina Foundation, 2011, p.13). 


Competency statements in the DQP 
are deliberately and relentlessly 
couched in ‘action verbs” that 
describe what students at particular 


levels should be expected to do. 
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This competency requires an assessment prompt asking for these things 
specifically and a response including each named element (different media 
or languages, actual citations in appropriate form, and actual comparisons 
of relative merit). An assessment prompt simply asking students to describe 
these elements or to select them from a presented list is not adequate. 

This suggests the importance of developing explicit tools for constructing 
assignments and determining the adequacy of student responses. 


Assignment Templates and Rubrics 

Throughout the DQP’s text, the authors emphasize that assessment should 
be embedded in courses throughout the curriculum. The primary vehicle 

or mechanism for determining whether or not students have mastered the 
competency, therefore, is a course assignment of some kind. To provide the 
necessary rigor, focus, and consistency for valid and reliable assessment, 
however, assignments used as assessments should have specific characteristics. 
Put simply, the construction of the assignment must unavoidably elicit 

a demonstration of the competency. Building such assignments can 
constitute a significant challenge, however, because most college faculty 
members are not explicitly trained to do this. As a result, too many faculty- 
made assignments or examination questions fail to elicit an appropriately 
configured student response. As an illustration of this, the hypothetical, yet 
typical, open-ended examination question “name and describe three aspects 
of the Renaissance” provides the student with almost no guidance about 
what a good answer should actually look like. “They wore funny clothes, they 
played funny instruments, and they sang funny songs” is an appropriately 
configured (although not very good) answer to the question posed. 


One way to avoid this difficulty is to identify the specific properties of an 
appropriate response and then to configure the assessment prompt so that the 
respondent has maximum information about what his or her response should 
look like. This is what an “assignment template” is designed to do. A growing 
number of institutions are using tools like this to ensure that all students 
approach the assigned task in a manner that is predictable and that allows 
them to fully demonstrate what they know and can do. For example, the 
Indiana University—Purdue University at Indianapolis (IUPUI) University 
Writing Center offers guides for such familiar kinds of assignments as 
preparing a literature review that synthesizes multiple sources.’ Similarly, 
West Coast University—a participant in the Lumina-funded WASC Senior 
Commission DQP project—is developing “signature assignments” to assess 
competencies contained in the DQP Intellectual Skills domain using an 
alignment matrix. 


‘The basic elements of an assignment template are intended to provide 
guidance about the specific characteristics of an appropriately configured 
answer to the posed topic or question. This includes 


TSee http://www. iupui.edu/-uwce/pdf/Literature%20Review%20and%20Synthesis.pdf. 
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of a given assignment to demonstrate 
mastery is to revisit the action verbs 
that describe the competency itself: 


13 


¢ = The central task that must be undertaken as well as the DQP domain 
and degree level in which it is located. For example, a central task to 
demonstrate DQP competencies in the realm of Analytical Inquiry 
might involve comparing and contrasting two or more arguments or 
points of view on a particular topic. 


° How the required task should be undertaken and the results 
communicated. For example, communications mechanisms noted 
in DQP competencies related to Quantitative Fluency include or question. 
verbal arguments, mathematical algorithms and constructs, and 
mathematical arguments using accepted symbolic systems. 


° How extensive or evidential the response should be. For example, DQP 
competencies listed under Communication Fluency and Use of 
Information Resources require two or more examples, two or more 
languages or media, and appropriate citations. 


Combining all three elements yields something like the following: “Compare 
the substance of [argument X] with [argument Y] by means of a written essay 
[of Z length] that cites at least three examples of important ways in which 
the arguments differ.” Realistically, an assignment or examination question 
should address not more than two or three competencies. Also, faculty 

will find it more comfortable to test for DQP competencies, which are by 
nature generic, in combination with more specific course-based knowledge 
and skills. Assignment templates of this kind can be constructed so that 

they cover content questions in virtually any discipline, as illustrated by the 
following examples* 


¢ Prepare an exhibit of not more than five discrete 2-dimensional 
pieces illustrating the range of chaos in color, drawing on at least two 
of the major color theory sources, e.g. Goethe, Kandinsky, Chevruel, 
in a 3-5 page catalogue of your exhibit. You are not required to 
present in the same 2-dimensional medium across all five pieces. The 
class exhibits will be displayed from April 1 - 30. It is now January 
15. [Associate's level, Broad integrative knowledge] 


¢ Suppose a new form of energy was developed that would emit 
no carbon, gases, of other pollutants. Critics of the development 
contend that within a month of its deployment, the earth’s rotation 
would slow from 24 to 26 hours per day. To guard against this and 
other consequences an Environmental Impact Statement must be 
prepared. In the space below, outline the chapters and subchapters 
of such a Statement. [Bachelor’s level, Applied Learning, integration, 
30-minute examination question] 


81 am indebted to Cliff Adelman and colleagues at Utah State University for these examples. 
An inventory of similar assignments and examination questions suited to assessing various 
DQP competencies at various levels are being collected in the “DQP Corner” of the NILOA 


website at http://www.learningoutcomesassessment.org/DQPCorner.html. 
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[The student is provided a diagram of a cell not at division stage 
with various structures labeled.] Describe the cell in terms of a) 
its current stage, b) its morphological signs of activity, and c) the 
structure that addresses the formation of its nuclear envelope. 
[Associate's level, Specialized Knowledge, translation of medium, 
examination question] 


Creationists maintain that the Second Law of Thermodynamics 
supports their point of view by positing that the natural direction 

of change is from complexity to simplicity. One implication of this 
is that human life—a complex form—could not have “evolved” 
from more primitive forms of life—and, consequently, would have 
had to have been created. Critique the reasoning of this position by 
succinctly presenting each of its points in logical order, together with 
arguments for and against. In a final paragraph, identify and describe 
the type of argument creationists are making: Is it evidential, 

logical, analogical, declarative, or some other form? [Bachelor’s level, 
Specialized Knowledge, Analytical Inquiry, short essay assignment]. 


You are given a map of the United Kingdom with three (3) airfields 
marked. You are flying a military interceptor aircraft with the 
following specifications (weight, fuel capacity, current fuel level, 
fuel use in different maneuvers), your location at point X, your 
current speed, the current reading of your fuel guage, the location 
of a refueling tanker at point M, its current speed, and the rate/ 
time of refueling. You are told that an alien aircraft is approaching a 
northeast coast radar station at a speed of Y and is currently located 
at Z. Is it 3 p.m. and the weather is closing. You are instructed to 
intercept the approaching aircraft, destroy it with missiles you are 
carrying, and return. At which airfield will you land? at what time? 
and how much fuel will you have left (the amount must be at or 
above 500 kg)? For each of these questions, present a symbolic 
formula that reflects the way you arrived at your solutions. Set a 
timer, and also indicate how long it took you to arrive at all those 
answers. All your responses should fit on one page. [Bachelor’s level, 
quantitative fluency] 


[The student is given a walking route map of the Lower East Side of 
New York City. Students are asked to walk the route and complete 
the following assignment within at least two weeks.] Prepare a 

series of short statements about what you see, the connections of 
what you see to the social and political history of the area, and your 
own analyses of the successes and failures of attempted changes. 


Specifically, 
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a) Identify the structure that most clearly illustrates the 
concept of “invasion and succession,” and present 
hypotheses about the dominant activities/responsibilities of 
those who used the structure at each phase. Describe who 
used the structure, in what sequence, and how. 


Assignment templates can be 
constructed so that they cover content 
questions in virtually any discipline. 


b) Illustrate the effects of the Tenement Laws of the late 19% 
century through citations of specific structures and their forms. 
As you do so, identify the origins and stimuli of these laws. 


c) Identify the structures and physical conditions toward 
which the Public Health Laws of the late 19" century are 
directed. Provide evidence that these conditions were either 
eradicated or still exist. For those that still exist, analyze and 
present the conflicting powers or interests that will affect 
the extent to which they can be addressed. [Bachelor’s level, 
Civic Learning, integration, class project assignment] 


Too often, lack of precision characterizes faculty judgments about the quality 
of a student response. “I know it when I see it” is an all-too-common answer 
to queries about how faculty process, code, and evaluate student work. 
About 20 years ago, to remedy this situation and to harmonize the grading 
of student work, structured grading guides began to emerge in higher 
education, although for decades prior to this the K-12 testing industry had 
been using such methods to score students’ written work.” Now known 

as “rubrics,” these guides typically provide detailed descriptions of specific 
attributes of the student work along several dimensions. 


A rubric, in many ways, is the mirror image of an assignment template. For 
example, if the latter prescribes a response with “at least three examples” 

(as in the assignment template above), the associated rubric will reflect this 
prescription by awarding a full score for a response that, indeed, has three 
examples and partial scores for responses that have fewer. Another dimension 
of the rubric would simultaneously enable the scorer to evaluate the quality 
of any comparison of two arguments that a response provides. A third might 
provide a metric to evaluate components of the written essay itself including 
its length, the sophistication and relevance of its analysis, and the consistency 
of its language with standards of academic discourse. Faculty at IUPUI and 
Ivy Tech Central Indiana are using just such a process to build common 
assessment rubrics aligned with the DQP for use in a transferrable general 


education initiative that is part of a statewide effort in Indiana, and as part of 
a Lumina-funded DQP demonstration grant to AAC&U. 


Tools for evaluating the quality of naturally occurring student work like 
assignment templates and rubrics are not easy to construct and take a good 


°One of the most popular of these is the “primary trait” grading approach (Walvoord & 
Anderson, 1998). 
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deal of training to apply effectively. They are essential, however, to actualize 
the approach to assessing student learning outcomes implied by the DQP 
necessarily embedded in an institution’s curriculum or co-curriculum. 


Navigating the Curriculum 

In the text accompanying the DQP, the authors make many points about 
what an appropriate college curriculum designed to foster the development 
of these competencies should look like. More particularly, we believe 

that the courses constituting the curriculum should be intentional and 
cumulative and, further, should feature many connections demanding and 
developing these competencies in different settings. Course sequencing, 
therefore, is critical—to ensure that the series of courses a given student takes 
includes successive benchmark assessments that build toward culminating 
demonstrations of mastery. In this way, adopting the DQP makes assessment 
the centerpiece of any institution’s curriculum, rather than simply one more 
feature of the courses that comprise it. This particular point of departure— 
that navigating the curriculum should contribute to an ongoing and 
cumulative vector of learning for each student—has implications for both 
curricular design and student advising. 


student work. 


Applying the DQP first makes curricular design more intentional by 
emphasizing the integration of assessment into planned course sequences. 
As I described earlier, curriculum maps are commonly used by colleges 

and universities to identify where in a particular set of courses a given 
competency is required, taught, or determined through assignments. Such 
application, of course, is after the fact. But similar curriculum maps can 
also be used up front as an aid to planning and implementing a yet-to-be 
developed curriculum. For example, as a specialist graduate-only institution 
in psychology, Sophia University intends to move its curriculum toward 
undergraduate programming. Through its participation in the WASC 
Senior Commission DQP demonstration project funded by Lumina, the 
institution has adopted the bachelor’s level competencies of the DQP as the 
foundation of its new undergraduate programming and reports that doing 
so allowed much quicker progress than modifying an existing program. 

A similar “ground up” curriculum development effort within the WASC 
Senior Commission group is being undertaken by Golden Gate University in 
completely redesigning its current undergraduate degree offerings consistent 


with the DQP. 


Taking the vector of student growth and development on each DQP 
competency as the primary point of departure, in place of the more 
familiar standpoint of content coverage, is a far more deliberate approach 
to curricular design than what customarily occurs. Indeed, in the case 

of Western Governors University (WGU), the sequence of assessments 
documented in this mapping process essentially defines the curriculum.'° 


'OWGU is not part of a Lumina-funded DQP demonstration project, but I mention it 
because it is currently the best example of a postsecondary institution constructed on a 
competency basis from the ground up. 
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Most institutions, of course, do not have WGU’s luxury to design their 
curriculum on an outcomes basis from the outset. However, by deliberately 
constructing curriculum maps to document course/assessment coherence 
after the fact and by creating more forward-looking revisions of these 

maps to redesign future course content and sequencing, any institution 

can increase the likelihood of its students acquiring the requisite skills and 
competencies. 


Course sequencing ts critical to ensure 


that the series of courses a given 
student takes includes successive 
benchmark assessments that build 
toward culminating demonstrations 
This way of thinking about a curriculum also enables the adoption of a of mastery. 

quite different approach to student transcripts in which—rather than the 

conventional list of courses the student has taken—transcripts are a record of 

what the student has mastered in what order and at what level. Some of the 

more sophisticated assessment management systems (see below) provide just 

such a record of student mastery. 


Used appropriately, the DQP also has significant implications for student 
advising. With a DQP-generated map of their curriculum, students can see 
more clearly what they are to learn and what will be expected of them at each 
point along their learning journey. Making prerequisite structures transparent 
and showing how what is learned and assessed at point X will be applied and 
tested at point Y helps head off one of the most common questions posed by 
students considering an institution's general education requirements: “Why 
do I have to take this class?” Showing how the DQP is enacted throughout 
the curriculum not only provides students with sound advice; it enhances 
their motivation and self-confidence as well. Considerable research supports 
the conclusion that students learn better when they clearly understand the 
learning expectations of them (Bransford, Brown, & Cocking, 1999). 


Documentation 

In a DQP context, assessment is ongoing and decentralized. It occurs 

every time a faculty member examines a particular student response 

toa posed examination question, demonstration, or assignment—so 
assessment is happening all the time. Because of this, the DQP approach 
requires a comprehensive record-keeping system for posting, housing, 

and manipulating data on what students have learned. As more than a 

few colleges and universities pioneering approaches like course-embedded 
assessment and portfolios discovered in the mid-1980s, the paper-and-pencil 
scoring approach with its myriad associated filing cabinets filled with student 
artifacts became so cumbersome it was dropped. 


An electronic record-keeping system of a kind appropriate for assessment 
within the DQP context resembles a conventional student record system, but 
is structured around competencies rather than courses as the unit of analysis. 
A typical entry for a particular student consists of the following elements 


¢ A faculty-assigned level of attainment on a particular competency, 
using an identified scale (for example, beginning, intermediate, and 
culminating); 
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e Ata particular point in time (for example, the end of spring 
term 2013); occurring in a particular setting (for example, a final 


examination essay in Biology 302); Ale DOP ee eratedim apes tier 
curriculum, students can see more 


e As assessed by a particular method (for example, a rubric associated 
with the particular competency). 


To create these entries, the system needs to have an electronic environment 
in which the faculty member can pull up the student work to be scored, as 
well as the rubric for the particular competency against which the work will 
be evaluated, and can record the resulting score (or scores) that she or he 
assigns. The system also needs to be linked with the regular student records 
system so that particular attributes associated with the student (for example, 
demographics, year of study, major discipline, prior assessment results, and 
so on) can be retrieved. This allows analytic reports for particular types of 
students to be generated. A parallel link with course records enables analyses 
of performance by discipline, class level (e.g., upper or lower division), course 
type, and delivery mechanism (for example, face-to-face lecture/discussion, 
online asynchronous, problem-based, internship/practicum, and so on). 
This also requires the system to have the capacity to generate reports ranging 
from disaggregated analyses of students’ performance of a competency by 
subpopulation to individual “mastery transcripts” showing what level each 
student has attained on each competency at a particular point in time. 


Many commercial assessment support packages have the requisite features. 
Examples include eLumen, TracDat, WEAVE On-Line, LiveText, 
TrueOutcomes, or Tk20. Some campuses, like Brandman University, in 

the WASC Senior Commission DQP demonstration project, are using the 
“turn it in” function of an electronic course management system (in this case, 
Blackboard) to upload student work for evaluation and the system’s rubric- 
based grading tool (“GradeMark”) to score student responses. 


Benchmarking and Comparison 

The DQP goes beyond simply establishing a particular set of competencies 
for an individual institution toward doing so across multiple colleges 

and universities. The ultimate vision, as illustrated by the qualifications 
frameworks established in other countries, is a set of standards describing the 
explicit meaning of degrees as national reference points on a limited set of 
carefully delineated competencies. 


A first challenge that frequently arises as institutions try to “adopt” the DQP 
is the relationship between its array of competencies and the institution’s 
own statements of student learning outcomes, usually developed over many 
years, which contain a host of embedded assumptions and compromises 

but which institutions are hesitant to give up in favor of a whole new set of 
competencies their own faculty members have not had a hand in developing. 
Under these circumstances, frequently, the institution's initial step is to map 
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the DQP onto its own student learning outcomes statements to ascertain the 
extent of overlap and to determine if anything is missing. This may uncover 
areas of distinction, like ethics or collaboration skills, where the institution’s 
own goals address areas not in the DQP. In most cases, though, institutions 
discover that the coverage of the DQP parallels that of their own statements. 
Because of this, multiple institutions can confidently benchmark their 
outcomes to the common reference point of the DQP even though their 


An electronic record-keeping system 
of a kind appropriate for assessment 
within the DQP context resembles a 
conventional student record system, 


individual outcomes statements continue to differ from one another. but is structured around competencies 
rather than courses as the unit of 
In this way, the DQP acts first in a benchmarking function as a kind of analysis. 


“universal translator” with respect to learning outcomes across a range 

of diverse institutions. For example, Bethel University and Holy Names 
University—participants in the CIC DQP project—used the DQP as a 
mechanism for improving internal communications among faculty who 
teach general education courses and to better integrate faculty advisory 
groups associated with general education. Meanwhile, Berry College used it 
to help foster better communication between academic affairs and student 
affairs personnel. Among HLC DQP project participants, Illinois College 
reports that discussions among different campus constituencies framed 
around the DQP helped diverse academic and co-curricular programs focus 
more intentionally on the overall campus experience. AAC&U and Oregon 
DQP project participant Portland State University also noted that DQP- 
based discussion is helping foster cross-department communication and 
alignment. In the WASC Senior Commission project, finally, The Master’s 
College is using the DQP to promote more explicit communication and 
collaboration across a range of campus groups to highlight the role of the co- 
curriculum in developing appropriate bachelor’s level competencies. 


As noted earlier, in the section on curricular mapping, simply translating 
one set of competency statements into another by way of the DQP is not 
enough. Acting on the DQP’s assessment implications requires an array of 
assignment templates and rubrics that can be used consistently and reliably 
across settings—and there are many different ways this can be accomplished. 
Beginning with the most complex, institutions have used the following 
approaches to achieve alignment and to boost the reliability of obtained 
results. 


¢ Multiple Third-Party Raters. Under this approach a number of raters 
examine a particular student artifact against a previously developed 
rubric and then reconcile their results. Raters may be drawn from 
different institutions or different academic units within the same 
institution. Alternatively, they might be constituted as a third-party 
review panel of content experts selected especially for this purpose." 


"Members of the program advisory committees that are frequently recruited by professional 
and vocational programs can be ideal for this purpose. 
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Artifacts, in turn, can be student papers, answers to a posed 
examination question or assignment, observed performances in a 
professional or capstone setting, or portfolio entries. This approach 
is likely to provide the most valid and reliable results but is likely to 
be expensive because it is so labor intensive. 


°  Cross-Rating. Under this approach, faculty members at a given 
institution arrange with counterpart faculty members teaching the 


The ultimate vision, as illustrated 

by the qualifications frameworks 
established in other countries, is a set 
of standards describing the explicit 


meaning of degrees as national 


same course with the same assignment at another institution to rate reference points on a limited set of 
each other’s student artifacts using a previously developed rubric. carefully delineated competencies. 


They then can discuss results to determine consistency and draw 
out any implications for improvement. This approach provides 
some degree of inter-institutional external validation with only the 
additional costs associated with the exchange’s logistics. 


¢  Multi-Institutional Rating. Under this—most common—approach, 
faculty members at a number of institutions use the same rubric to 
rate similar student artifacts independently at their own institutions. 
In this case, external validation is obtained by disciplined use of 
the rubric, by substantial prior training, and by prior collective 
applications of the rubric to different kinds of student work. 


Although most examples of the use of rubrics to examine student work in regular 
classes or assembled in portfolios involve individual faculty members acting alone, 
some examples of externally validated applications of rubric-based scoring are 
beginning to emerge. One prominent example is a multicampus initiative currently 
being undertaken under the auspices of the Massachusetts Board of Regents using 
rubrics developed by AAC&U (Ewell, 2013). Under this initiative, teams of faculty 
members are using the Valid Assessment of Learning in Undergraduate Education 
(VALUE) rubrics, developed by AAC&U, to rate student artifacts on selected 
assignments in an aligned fashion across seven state public universities and fifteen 
community colleges. In this initiative, scoring alignment is being achieved through 
ongoing training and selective double-scoring of student responses. 


Conclusion 

As these examples illustrate and as I emphasized at the outset of this paper, 
engaging assessment in the context of the DQP requires faculty to be much 
more systematic and intentional than is currently the case at most colleges 
and universities. From the standpoint of curricular and course design, 
considerable planning and attention is needed to ensure that the appropriate 
competencies at the proper levels are developed or demanded across course 
sequences. From the standpoint of assessment design, assignments and 
associated rubrics must be carefully scripted to elicit the proper kinds of 
student responses and to judge their adequacy. Both standpoints also require 
a great deal of collaboration within and across campuses as well as collective 
ownership of the undergraduate teaching and learning process. Because 
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this is not the prevalent culture among American higher education faculty, 
implementing these ideas at scale will require considerable investments in 
faculty development. These faculty development efforts, moreover, must 
be premised on an important shift of perspective. The DQP asks faculty 
members to examine the entire instructional process from the inside out— 
starting from the perspective of learners and what they learn instead of the 
perspective of teachers and what they teach. 


Finally, the assessment philosophy and approaches implied by the DQP 
described above allow institutions to transcend the much-discussed dilemma 
of assessment for accountability as opposed to assessment for improvement 
(Ewell, 2009). Because assessment results are high stakes in the sense that 
they certify (rather than merely indicate) particular levels of attainment 
against a common and publicly established set of standards, these approaches 
provide built-in accountability. At the same time, assessment results amassed 
and examined over time can reveal observable patterns of strength and 
weakness in particular competencies or for particular kinds of students that 
can inform improvements in curriculum or pedagogy at any level. 


At this point, we are just beginning to see how far the transformations in 
teaching, learning, and assessment implied by the DQP will go and how 
much of their potential can be realized. The Lumina-funded follow-on 
projects launched earlier this year now involve roughly 220 institutions 
and, as indicated by the examples cited here, are already suggesting different 
ways forward. My purpose in this brief paper is to provide a few of their 
experiences as a “tasting menu,” while encouraging a// institutions to 
experiment and communicate. If we do this together, we can learn a great 


deal. 


The DQP asks faculty members to 
examine the entire instructional 
process from the inside out—starting 
from the perspective of learners 

and what they learn instead of the 
perspective of teachers and what they 
teach. 
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Afterword 


The DQP and the Assessment Challenges Ahead 


Carol Geary Schneider 


Peter Ewell’s paper provides a timely and much-needed guide both to the 


Degree Qualifications Profile (DQP) and to the assessment challenges that We will need an organized and 
the DQP presents. He accurately describes the kinds of assessment that the strategic campaign if colleges 
DQP promotes: faculty judgments of student performance using standards and universities are to be more 
that challenge students to reach and demonstrate high levels of authentic intentional and more effective 
accomplishment. in graduating students who are 


demonstrably well prepared for 
work, for civic responsbility, and for 
realizing their hopes for a better life. 


But Ewell’s essay may be a bit too logical, systematic, and reasoned for this 
fraught and overburdened moment in higher education history. My purpose 
here is to discuss directly and candidly the uphill battle that the DQP faces 
to promote more intentional learning and richer forms of demonstrated 
student achievement.' I believe that we will need an organized and strategic 
campaign, not just good assessment tools, if colleges and universities are 

to be more intentional and more effective in graduating students who are 
demonstrably well prepared—for work, for civic responsibility, and for 
realizing their hopes for a better life. 


Specifically, this campaign will need to strongly affirm the DQP’s clear focus 
on assessment of students’ authentic work as the best way to strengthen 
students’ college learning and to document the quality of their achievement. 
Concurrently, the campaign will need to directly refute knee-jerk 
assumptions that the DQP invites an increase in standardized testing at the 
college level. It does not. 


As with Ewell, I write as one of the DQP authors, but also as someone 

who heads a national organization that has assiduously focused on learning 
outcomes for over a decade and that now, under a Lumina grant to AAC&U, 
is partnering with two- and four-year campuses in nine states to test ways of 
assessing students’ DQP learning outcomes, with a focus on transfer. I also 
am working with educational leaders across the country and in Washington, 
DC, speaking in many contexts about the DQP, reading reports on other 
organizations DQP experiments, and, above all, listening with increasing 
concern to what my colleagues have to say. 


In a nutshell, what we are learning is that the while the DQP was designed 
to be a promising beacon for transformative change, a seemingly growing 
number of influential leaders do not see it that way. In addition, the DQP 
faces very real challenges on campus, especially when it comes to assessment. 
This is not for lack of conscientious commitment by faculty members 
working to operationalize the DQP on campus, but rather because the 

level of educational intentionality and collaboration implied in the DQP 


'The current version of the DQP is being beta-tested by about 220 colleges, universities, and 
community colleges, the majority working under Lumina grants, but about 60 at last count 
simply trying it on for size. The DQP will be revised and offered in a second edition informed 
by campus experience and advice. 
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assessment principles contrasts so much—as Ewell’s essay gingerly points 
out—with deeply-rooted campus norms. What I think we are seeing is that 
the habit of treating college learning as a set of separate, discrete, and even 
“siloed” units—individual courses, the majors, general education, the co- 
curriculum, and so on—works at cross-purposes to the DQP’s conception of 
a more intentional and, ultimately, integrative educational experience. 

If higher education is to move beyond the DQP’s “bold concept” stage to 
realize its intended purpose—that colleges and universities adopt a shared 
framework to guide student progress through college and to promote new 
and more authentic approaches to assessing learning—we must address two 
key issues: 


The DQP was designed as a clarion 
call to move faculty judgment and 
students’ own work back to the center 
of assessment and accountability. 


¢ The fear of what comes next if higher education signs on—both to 
the idea of a shared degree framework and to the DQP itself. 


¢ The danger that the “sudivide and get it done” approach to DQP 
experiments may obscure the DQP’s intended role as a catalyst 
for fostering and assessing students’ cumulative and integrative 
learning. 


Fearing What Comes Next 

Let’s deal with the fear first. Many leaders—especially those at research 
universities—both assume and worry that once the DQP is established—let’s 
say, through accrediting organization endorsement and/or by state systems 
adopting it—the next thing to follow will be imposition of DQP-calibrated 
standardized test(s). 


The logic of resisters is straightforward, based directly on what transpired 

in the K-12 system. First came “Common Core Standards,” then the 
development of standardized tests to measure student progress against 

the standards. Resisters further point out that Lumina Foundation for 
Education, even as it supports a variety of campus-based DQP experiments, 
is also funding (with others) the development of standardized tests for 
college-level learning across the globe.’ It is not much of a stretch to assume 
that if Lumina—in concert with ministries of education in other countries 
and the U.S. Department of Education as well—is promoting the DQP 
and also investing in international tests that almost surely will be used for 
international rankings, then it is only a matter of time until the foundation 
brings these two wings of its enterprise together. 


But the DQP is orthogonal to standardized tests. Indeed, as Lumina’s senior 
leaders readily affirm, the DQP was designed as a clarion call to move faculty 


judgment and students’ own work back to the center of assessment and 
accountability. The centerpiece of DQP assessment, as Ewell makes clear, is 


> Source: ahelo@oecd.org 
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supposed to be the assignments that faculty develop and give to students. 
The DQP seeks to change what counts as the most important evidence of 
student competency, away from the standardized metric and toward the 
work students do on nonstandard tasks. 


College must prepare learners to deal 


with the complex and uncertain, not 


To clarify and to ensure that this intended change prevails, Lumina and those just with the rote and routine, 


involved in the DQP effort need a communications and cultural change 
strategy, not just an assessment strategy, that says plainly, repeatedly, and 
insistently: The DQP promotes transformative change in what counts as 
evidence of student learning and in mainstream strategies for developing and 
documenting student achievement. 


Its fundamental purpose is preparing students to tackle nonstandard, 
unscripted problems and questions. Unscripted problems, by definition, 
are those where “right answers” are not known and where the nature of 
the problem itself is likely uncertain at best, and often actively contested. 
Standardized testing is antithetical to assessing the adaptive and inventive 
competencies that are the core of the DQP. This is because the available 
standardized tests often disguise rather than illuminate what students can 
actually do with nonstandard problems. Yet nonstandard problems are 
the ultimate test of students’ competence—at work, at life, and in the 
community. 


Health care “reform” in the U.S. offers an instructive illustration of the 

kind of adaptive and context-attentive learning that the DQP seeks to 
promote—and to assess. What seems “best” in health care reform depends 
on the questions and assumptions one brings to the debate, and individuals 
themselves may need to address these issues both against the tangle of 
passionate but competing public disputes and also in the context of potential 
implications for self and loved ones. 


College must prepare learners to deal with the complex and uncertain, not 
just with the rote and routine. Assessments ought to show how well students 
can integrate context, inquiry, evidence, applications, and implications. 
Multiple-choice tests will not meet this standard. 


For all these reasons, the performance-based tasks faculty assign to 
students—in their courses and field-based learning—must be the 
centerpiece of 21 century quality assurance in terms of student learning. 
These performance-based tasks need to give students collaborative practice 
in addressing complex questions. They also provide the best evidence of 
students’ growing proficiency both in analyzing significant problems and in 
applying their learning in field-based as well as academic settings. 


Ewell cogently described the steps involved in assessing the DQP 
competencies, with students’ work as the required evidence both of their 
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achievement, and, ultimately, of their meeting the expected qualifications for 

a degree. But make no mistake. Unless a campus moves students’ actual work 
to the center of the DQP endeavor, there’s really not much use in adopting an 
outcomes framework—whether it’s DQP or the Essential Learning Outcomes 
advanced in AAC&U’s LEAP initiative or another variant. 


In sum, to promote the kind of learning today’s students need, those 
advocating using the DQP must make the case that it is a transformative 
framework for 


¢ Promoting far-reaching change in what counts as the primary 
evidence about students’ learning gains in college; 


¢ Foregrounding performance-intensive assignments, anchored in 
academic and pre-professional studies, through which students 
practice, develop, and demonstrate the competencies they need to 
address nonroutine problems in work, civic participation, and their 
personal lives—all of which is beyond the capacity of standardized 
tests as currently constructed; 


¢ Putting faculty judgment back at the center of efforts to strengthen 
student learning and foster educational excellence; 


¢ Fostering integration and application of learning—from multiple 
courses and contexts—as the best evidence of competency and the 
best way to prepare students to grapple with new and unscripted 
problems. 


Fears are not easily dispelled. DQP leaders need to have the courage of our 
shared commitments to lead transformative change for 21* century learners. 
That courage will require us to explain why, in a global era marked by 
dizzying complexity and volatility, we need to create forms of assessments in 
which standardized tests are no longer the center of the accountability action. 


Subdivide and Get It Done: 

The Fragmentation of Campus DQP Experiments 

The second obstacle confronting the transformative potential of the DQP is 
the “subdivide” to “get-it-done” strategy that most of the DQP assessment 
experiments have adopted. (This same observation applies to the DQP 
curricular projects as well.) The problem is that this strategy may have the 
unintended consequence of obscuring the DQP’s intention to frame a more 
integrative and cumulative design for college learning. 


In order to document the outcomes described in the DQP, many campuses 
experimenting with the DQP have focused their assessment work on a 
specific subset of the curriculum, such as the business or biology program in a 


DQ? leaders need to have the courage 
of our shared commitments to lead 


transformative change for 21” 
century learners. 
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transfer context, or writing in general education, or quantitative reasoning in 
four or five disciplines. These are “subdivide assessments,” meaning that they 
tackle only a subset of the five DQP areas of learning and, in the case of the 
Intellectual Skills area, take on only a couple of the five complex Intellectual 
Skills—analytic inquiry, communication fluency, quantitative fluency, 
engaging diverse perspectives, and use of information resources—for which 
the DQP maps out illustrative competencies. 


‘The subdivide assessment strategy is attractive as it allows faculty and staff 
to connect the DQP with real questions or problems that the experimenting 
campuses want to address. This is a politically smart, understandable choice, 
and perhaps the only way that the DQP could be introduced at all, given 
the other issues with which all collegiate institutions are grappling such as 
cost-cutting, completion, and digital innovations. These challenges, along 
with “initiative fatigue’—due to the ever-growing number of state or 
system mandates or organization-sponsored innovations—are realities that 
educational leaders have to manage. 


Even so, wide use of this practical strategy—the commitment to “subdivide 
and get it done’—may mean that we will learn less than is needed from the 
various DQP assessment experiments under way. Moreover, it may limit the 
number of people actually working with the DQP. The faculty involved may 
be working only with a particular discipline—the “Specialized Knowledge” 
part of the DQP. Or they may be working only on general education, or 
even on ways to assess a particular skill such as writing that is seen as part of 
general education. 


The downside of a subdivide assessment strategy is that experimenting 
campuses tend not to deal with the DQP as a whole, either as an educational 
framework or “profile” or with efforts to enact the intended connections 
across the five DQP areas of learning designed into the DQP framework 
specifically to intentionally foster integrative and adaptive learning. Thus, 
the experiments may overlook the task of assessing the integrative learning— 
across fields of study or between analytic and applied learning—that the 
DQP design was intended to actively promote. 


To be specific, a 21" century postsecondary education must help students 
develop the capacity to see and make connections between broad and 


specialized knowledge and between knowledge and skills. Graduates must also 


be able to integrate knowledge and skill with the kinds of situated judgments 
(and new learning) required to apply their learning in different contexts and 
for civic inquiry and problem-solving. 


For these reasons, none of the five areas of DQP attainment is self-contained. 


Minimally, each of the five DQP areas of learning requires the integrated 
deployment of broad knowledge, field- or topic-specific knowledge, and 


When a campus assesses intellectual 
skills only within general education 
courses or a major, it misses an 
opportunity to explore how the 
curriculum overall can help students 
actually connect the different parts 
of their educational experiences and 
learning. 
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intellectual skills. Applied Learning and Civic Learning call for even more 
complex forms of integration, application, and—implicitly—for reflective 
and evaluative learning as well. 


Said another way, when a campus assesses intellectual skills only in general 
education courses, or knowledge and skill only within a major, it misses an 


To foster intentional and 


integrative learning, and to 


demonstrate its attainment through 


opportunity to explore how the curriculum overall (including field-based assessment, faculty collaboration 
learning) can help students actually connect the different parts of their across curricular boundaries is 
educational experiences and learning. Indeed, the subdivide strategy may indispensable to success. 


inadvertently signal to students that integrative learning is not necessary to 
prepare them for the complex and unscripted problems they will encounter in 
the workplace, the public sphere, and in their personal lives. Just the opposite 
is actually true! 


I also worry that, across all the DQP experiments, too much of the campus 
discussion is centered on “outcome list alignment,” whereby institutions line 
up their DQP outcomes with institutional or general education outcomes, 
state or community college system-level outcomes (which, when they exist, 
usually are described as general education outcomes), and specific program 
outcomes for major fields, such as business or engineering, that may also be 
influenced by outcomes specified by a program-specific accreditor. 


While aligning such outcomes is a necessary first step for campus leaders—if 
only to ask the question whether the DQP gets at issues the system, campus, 
or discipline considers priorities—there is a huge distinction between 
“outcome list alignment” and the kind of curriculum/competency/assessment 
mapping that Ewell sets forth in his guide to DQP assessment. 


Curriculum mapping asks whether the educational experience is 

designed to effectively foster the intended learning. In terms of the DQP, 
curriculum mapping ought to explore whether the educational experience 
was implemented well enough for students to demonstrably acquire the 
integrative and applied learning the DQP describes. 


Outcome list alignment, thus, is necessary but far from sufficient for the 
kind of integrative educational planning and assessment that the DQP 
outlines. The curriculum mapping Ewell describes so well in his paper is the 
critical key, both to the quality of DQP competency development and to the 
assessment of students’ cumulative and demonstrated achievement. 


To say the obvious, students learn what they practice. The DQP is a design 
for a hands-on, practice-rich, integrative education. To foster this kind of 
intentional and integrative learning, and to demonstrate its attainment 
through assessment, faculty collaboration across the usual curricular 
boundaries is indispensable to success. 
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Last Words 


One way to address both the fear and the fragmentation that threaten the 


current DQP experiments is to make sure that all the DQP projects view The DOR iaabold cHerecahels 


their particular initiatives—outcomes articulation, curriculum planning, 
assessment, transfer collaborations, and the like—as micro-explorations of 
the larger questions that the DQP poses to higher education: What kinds 
of learning will actually prepare students to contribute and thrive, both in a 
turbulent economy and in a globally engaged democracy? What practices— 
both in the curriculum and in off-campus sites for learning—best foster the 


higher education move beyond credit 
hours to competency and beyond 
fragmented learning to intentionally 
preparing students to integrate and 
apply their learning to unscripted 


intended competencies? How do we help students see the larger aims of their problems and responsibilities. 


education, and what would tell us best whether students have developed the 
ability to deal with unscripted, nonroutine problems? 


If every campus and organization now working with the DQP moves such 
discussions to the top of its agenda—for faculty, staff, stakeholder, and board- 
level debate—then it will become far more likely that leaders at all levels 

will actually engage in the kind of creative educational work—generative 
innovation—that the DQP was designed to elicit. 


The DQP is a bold effort to help higher education move beyond credit 
hours to competency and beyond the fragmented learning too many 
students experience to intentionally preparing students to integrate and 
apply their learning to unscripted problems and responsibilities. The DQP 
was deliberately written as a first-draft “beta” version, with revisions and 
improvements both anticipated and welcomed. 


Those involved with this bold transformative experiment need to ensure 
that the micro-experiments become connected, within each campus and 
across the participating experiments, so that the potential educational value 


of the DQP’s intended assessment strategy becomes both visible and widely 
debated. 


Many of us learned long ago that situated and collaborative reflections about 
what we are “learning by doing” are a critical and even indispensable key to 
both professional development and competence. The DQP experiments are 
mote likely to take root if integrative, reflective, and collaborative dialogue 
and learning become central themes in the next phase of this very important 
educational experiment. 


Carol Geary Schneider 


President, Association of American Colleges and Universities 


National Institute for Learning Outcomes Assessment 


29 


References 


Allen, M. J. (2004). Assessing academic programs in higher education. Boston, MA: Anker. 
Allen, M. J. (2006). Assessing general education programs. Boston, MA: Anker. 


Association of American Colleges. (1985). Integrity in the college curriculum: A report to the academic community. 


Washington, DC: Author. 


Association of American Colleges and Universities. (2008). Our students’ best work: A framework for accountability 
worthy of our mission (2"™ ed.). Washington, DC: Author. 


Bloom, B. S. (1956). Taxonomy of educational objectives: The classification of educational goals. New York, NY: 
Longman. 


Bransford, J. D., Brown, A. L., & Cocking, R. R. (1999). How people learn: Brain, mind, experience, and school. 
Washington, DC: National Research Council, National Academies Press. 


Bresciani, M. J., Gardner, M. M., & Hickmott, J. (2009). Demonstrating student success: A practical guide to outcomes- 
based assessment of learning and development in student affairs. Sterling, VA: Stylus. 


Driscoll, A., & Wood, S. (2007). Developing outcomes-based education for learner-centered education: A faculty 
introduction. Sterling, VA: Stylus. 


Ewell, P. T. (2002). An emerging scholarship: A brief history of assessment. In T. W. Banta & Associates (Eds.), 
Building a scholarship of assessment (pp. 3-25). San Francisco, CA: Jossey-Bass. 


Ewell, P. T. (2009). Assessment, accountability, and improvement: Revisiting the tension (NILOA Occasional Paper 
#1). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes 
Assessment. 


Ewell, P. T. (2013). Having it both ways: Massachusetts’ approach to statewide assessment. Assessment Update, 25(1). 


Halualani, R. T., Haiker, H., & Lancaster, C. (2010). Mapping diversity efforts as inquiry. Journal of Higher Education 
Policy and Management, 32(2), 127-136. 


Kuh, G. (2012, October 29). Degree Qualifications Profile: What is the DQP and what does it mean for assessment? 
Presentation at the 2012 Assessment Institute of the Indiana University—Purdue University Units of Planning 
and Institutional Improvement, Indianapolis, IN. 


Lumina Foundation. (2011). The Degree Qualifications Profile. Indianapolis, IN: Author. 

National Institute of Education. (1984). Involvement in learning: Realizing the potential of American higher education 
(Final report of the Study Group on the Conditions of Excellence in American Higher Education). 
Washington, DC: U.S. Government Printing Office. 


Walvoord, B. E., & Anderson, V. J. (1998). Effective grading: A tool for learning and assessment. San Francisco, CA: 
Jossey-Bass. 


National Institute for Learning Outcomes Assessment | 30 


NILOA National Advisory Panel 


Joseph Alutto 
Provost 


The Ohio State University 


Trudy W. Banta 

Professor 

Indiana University-Purdue University 
Indianapolis 


Wallace Boston 
President and CEO 
American Public University System 


Molly Corbett Broad 
President 
American Council on Education 


Judith Eaton 
President 
Council for Higher Education Accreditation 


Richard Ekman 
President 
Council of Independent Colleges 


Mildred Garcia 

President 

California State University - 
Fullerton 


Susan Johnston 
Executive Vice President 
Association of Governing Boards 


Stephen Jordan 
President 
Metropolitan State University - Denver 


Mary Kalantzis 
Dean, College of Education 
University of Illinois Urbana-Champaign 


Paul Lingenfelter 
President 
State Higher Education Executive Officers 


George Mehaffy 

Vice President 

Academic Leadership and Change 
American Association of State Colleges and 
Universities 


Charlene Nunley 

Program Director 

Doctoral Program in Community College 
Policy and Administration 

University of Maryland University College 


Kent Phillippe 

Associate Vice President, Research and 
Student Success 

American Association of Community Colleges 


Randy Swing 
Executive Director 
Association for Institutional Research 


Carol Geary Schneider 

President 

Association of American Colleges and 
Universities 


Michael Tanner 
Chief Academic Officer!Vice President 


Association of Public and Land-grant 
Universities 


Belle Wheelan 
President 
Southern Association of Colleges and Schools 


Ralph Wolff 
President 
Western Association of Schools and Colleges 


Ex-Officio Members 


Timothy Reese Cain 
Assistant Professor 
University of Illinois Urbana-Champaign 


Peter Ewell 

Vice President 

National Center for Higher Education 
Management Systems 


Stanley Ikenberry 
President Emeritus and Regent Professor 
University of Illinois 


George Kuh 

Director, National Institute for Learning 
Outcomes Assessment 

Adjunct Professor, University of Illinois 
Urbana-Champaign 

Chancellors Professor Emeritus, Indiana 
University 


NILOA Mission 


NILOA’s primary objective is to 
discover and disseminate ways that 
academic programs and institutions 
can productively use assessment data 
internally to inform and strengthen 
undergraduate education, and exter- 
nally to communicate with policy 
makers, families and other stake- 


holders. 


DQP 


The Degree Qualifications Profile was 
created by the Lumina Foundation 
in 2011. The DQP illustrates clearly 
what students should be expected to 
know and be able to do once they 


earn their degrees — at any level: 

the associate, bachelor’s and master’s 
degrees — which constitute the great 
majority of postsecondary degrees 
awarded by U.S. colleges and universi- 
ties — regardless of a student’s field 
of specialization. The Degree Profile 
describes five basic areas of learning: 
Broad, Integrative Knowledge; 
Specialized Knowledge; Intellectual 
Skills; Applied Learning; and Civic 


Learning. 


National Institute for Learning Outcomes Assessment | 31 


About NILOA 


¢ The National Institute for Learning Outcomes Assessment (NILOA) 
was established in December 2008. 


e NILOA is co-located at the University of Illinois and Indiana 
University. 


¢ The NILOA website went live on February 11, 2009. 
www. learningoutcomesassessment.org 


¢ The NILOA research team has scanned institutional websites, 
surveyed chief academic officers, and commissioned a series of occa- 
sional papers. 


¢ One of the co-principal NILOA investigators, George Kuh, founded 
the National Survey for Student Engagement (NSSE). 


e The other co-principal investigator for NILOA, Stanley Ikenberry, 
was president of the University of Illinois from 1979 to 1995 and of 
the American Council of Education from 1996 to 2001. 


¢ Peter Ewell joined NILOA as a senior scholar in November 2009. 


NILOA Staff 


NATIONAL INSTITUTE FOR LEARNING OUTCOMES ASSESSMENT 


Stanley Ikenberry, Co-Principal Investigator 

George Kuh, Co-Principal Investigator and Director 

Peter Ewell, Senior Scholar 

Jillian Kinzie, Associate Research Scientist 

Pat Hutchings, Senior Scholar 

Timothy Reese Cain, Senior Scholar 

Natasha Jankowski, Project Manager and Research Analyst 
Gianina Baker, Research Analyst 

Paul Myers, Research Analyst 

T. Jameson Brewer, Research Analyst 


Robert Dumas, Research Analyst 


NILOA Sponsors 


Lumina Foundation for Education 


The Teagle Foundation 


University of Illinois, College of Education 


Produced by Creative Services | Public Affairs at the University of Illinois for NILOA. 10.032 


National Institute for Learning Outcomes Assessment | 32 


National Institute for Learning Outcomes Assessment 


For more information, please contact: 


National Institute for Learning Outcomes Assessment (NILOA) 
University of Illinois at Urbana-Champaign 

340 Education Building 

Champaign, IL 61820 


learningoutcomesassessment.org 
njankow2@illinois.edu 

Fax: 217.244.5632 

Phone: 217.244.2155 


